[cfg, recipe, tests] feat: wire DAPO overlong reward buffer on V1 omni loop - #502

Summary

Implements Phase 2 of #446 (Qwen3-Omni Thinker DAPO): wires the overlong
response-length penalty into the V1 omni reward path.

Stacked on #456 (Phase 1) — not yet merged. This branch is rebased onto
wenzhe/qwen3-omni-thinker-dapo at bc0bee6 (current #456 head), so the diff
against main includes #456's commits. Please review only the two commits on
top (feat: wire DAPO overlong reward buffer on V1 omni loop (#446 Phase 2),
chore: trim redundant comments) until #456 lands; I'll rebase and shrink the
diff once it merges.

2026-09-04 update: rebased onto #456's latest head (it had moved since
this PR was opened — the Phase-1 GPU smoke script was removed there per
review feedback, since the existing GSPO smoke covers the same pipeline).
Phase 2 restores that smoke script under its own scope, since it's this PR's
own e2e coverage for the overlong buffer, not a reintroduction of the removed
Phase-1 smoke. CPU tests pass post-rebase; the GPU e2e run below predates the
rebase and should be re-run before merge — I don't have further Modal/GPU
access in this pass.

What's actually wired

verl's registered dapo reward manager
(verl.experimental.reward_loop.reward_manager.dapo.DAPORewardManager) already
reads config.reward.reward_kwargs.overlong_buffer_cfg and
config.reward.reward_kwargs.max_resp_len — but verl_omni's reward.yaml
never declared a reward_kwargs key, so any
++reward.reward_kwargs.overlong_buffer_cfg.enable=true override on the CLI
was silently swallowed rather than reaching the manager. This PR:

• Adds reward.reward_kwargs (overlong_buffer_cfg.{enable,len,penalty_factor,log},
  max_resp_len) to verl_omni/trainer/config/reward/reward.yaml, disabled by
  default, and regenerates the four _generated_*.yaml trainer configs via
  scripts/generate_trainer_config.sh.
• Enables overlong shaping in the DAPO tiny-random smoke test
  (tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh), sized
  against its own data.max_response_length=512. The naive-manager AVQA
  example (examples/dapo_trainer/qwen3_omni/run_qwen3_omni_thinker_dapo_lora_v1.sh)
  is untouched and documented as such — overlong shaping is a no-op under
  reward.reward_manager.name=naive.
• Adds tests/utils/test_dapo_overlong_reward_on_cpu.py, which drives
  DAPORewardManager directly (no Hydra/trainer bring-up) and asserts the
  overlong penalty actually changes the reward on a truncated dummy response
  — this is Phase 2's exit criterion from the issue.
• Splits the shared launcher-contract assertions in
  tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py so "no overlong
  config" only applies to the naive-manager example, and adds
  overlong-specific assertions for the dapo-manager smoke test.

Not in scope: dynamic sampling (Phase 3), and any reward-loop metrics
plumbing beyond what DAPORewardManager already logs into reward_extra_info
(overlong, overlong_reward).

Test plan

CPU (all green locally):

python3 -m pytest tests/utils/test_dapo_overlong_reward_on_cpu.py tests/utils/test_qwen3_omni_dapo_launcher_on_cpu.py -q

GPU e2e (2-GPU tiny-random smoke, exercises the new reward.reward_kwargs.*
overrides end-to-end through the real V1 trainer loop):

NUM_GPUS=2 TOTAL_TRAIN_STEPS=2 bash tests/special_e2e/run_dapo_qwen3_omni_thinker_lora_v1_smoke.sh

Ran this on a Modal 2×A10G box against an improvised environment (CUDA
12.4.1 base + the exact pins from .github/actions/gpu-smoke-prepare/action.yml),
since I don't have access to CI's private cu130 registry image.

First two attempts hit environment issues unrelated to this PR: a missing
apt package, then a genuine upstream bug in flashinfer-python==0.6.16.post3
(pulled in transitively via vllm==0.27.0, no floor pinned anywhere in this
repo) that crashes any TP>=2 GPU run at import time. Filed as #505, fixed
in #506 (floors flashinfer-python>=0.6.16.post4 in the gpu extra) — see
those for the full root-cause writeup.

With flashinfer-python==0.6.16.post4 pinned (an environment-only override,
nothing in this repo changed), the smoke test passes:

RETURNCODE 0
Qwen3-Omni Thinker DAPO+LoRA e2e V1 smoke without dynamic sampling passed.
Training Progress: 100%|██████████| 2/2 [01:54<00:00, ...]

Per-sample DAPO reward scores varied across the batch (-1.0, -0.3438,
-0.0625), consistent with the overlong-buffer penalty grading responses
approaching/at max_resp_len=512 (buffer len=128, so penalty onset at
~384 tokens, full -1.0 at 512).

One caveat I'm not glossing over: I could not find an explicit
step:N - reward/overlong... console-logger line in the captured output —
grepped the full stdout+stderr for step: and got zero matches for any
metric, not just overlong, so this looks like a console-logging/capture gap
in this V1 path rather than evidence the wiring is broken (the reward
computation itself clearly executed through the DAPO path, per the reward
scores above). Worth a maintainer's eyes on whether metrics are expected to
print to console here or only to the configured tracker backend.

AI usage disclosure

Claude Code was used to trace the config-plumbing gap, implement this change,
and write the tests described above. A human submitter (onepunchmonk) has
reviewed every changed line and will review the Modal GPU run before taking
this out of draft.